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Mr.  Chairman  and  Members  of  the  Subcommittee: 


Results  in  Brief 


I  am  pleased  to  be  here  today  to  discuss  issues  the  Department  of  Defense 
(DOD)  faces  in  its  acquisition  of  weapon  systems  and  the  application  of 
best  practices  to  help  address  those  issues.  With  DOD’s  annual  research, 
development,  and  production  spending  for  weapon  systems  at  about  $85 
billion — coupled  with  suggestions  from  within  DOD  and  the  Congress  that 
it  should  be  substantially  higher — the  Subcommittee’s  oversight  of 
acquisition  policy  can  have  a  major  impact  on  the  value  the  taxpayer  gets 
for  that  expenditure. 

After  having  done  hundreds  of  reviews  of  major  weapon  systems  over  the 
last  20  years,  we  have  seen  many  of  the  same  problems  recur — cost 
increases,  schedule  delays,  and  performance  problems.  At  your  request, 
we  have  undertaken  a  body  of  work  that  examines  weapon  acquisition 
issues  from  a  different,  more  cross-cutting  perspective — one  that  draws 
lessons  learned  from  the  best  commercial  product  development  efforts  to 
see  if  they  apply  to  weapon  system  development.  In  the  past  few  years, 
leading  commercial  firms  have  developed  increasingly  sophisticated 
products  in  significantly  less  time  and  at  lower  cost.  These  firms  include 
the  Boeing  Commercial  Airplane  Group,  Bombardier  Aerospace, 
Caterpillar,  Chrysler  Corporation,  Ford  Motor  Company,  Hughes  Space 
and  Communications,  Motorola  Corporation,  and  3M.  Our  work  shows 
that  valuable  lessons  can  be  learned  from  the  commercial  sector  and  can 
be  applied  to  the  development  of  weapon  systems. 

My  testimony  focuses  on  how  best  commercial  practices  help  shape 
improvements  in  the  way  DOD  operates  and  makes  decisions  through  the 
weapons  acquisition  process.  Specifically,  I  will  discuss  why  weapon 
system  acquisition  problems  persist  and  how  a  process  based  on  best 
practices  can  lead  to  better  weapon  system  outcomes.  These  issues  have 
significant  implications  for  decisions  made  on  individual  programs  and  for 
DOD’s  larger  modernization  goals,  which  I  will  also  discuss. 


The  pressures  of  competing  for  the  funds  to  launch  and  sustain  a  weapon 
system  program  create  incentives  for  starting  programs  too  early; 
overpromising  performance  capabilities:  and  understating  expected  costs, 
schedules,  and  risks  associated  with  developing  and  producing  the 
weapon.  Leading  commercial  firms  have  adopted  a  knowledge-based 
approach  to  developing  new  products  that  embodies  incentives  that 
encourage  realism,  candor,  and  meeting  product  expectations.  Making 
sure  that  new  technology  is  mature — that  is,  demonstrated  that  it  works — 
is  the  foundation  for  this  approach.  DOD’s  variances  from  best 
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commercial  practices  result  in  higher  costs,  compromised  performance, 
and  questionable  cost  benefit  approaches.  A  knowledge-based  approach 
can  be  used  to  reshape  DOD’s  acquisition  process.  By  itself,  such  a 
process  will  not  produce  better  program  outcomes  unless  it  influences  the 
decisions  made  on  individual  weapon  systems.  If  a  knowledge-based 
acquisition  process  is  put  in  place  and  used,  there  are  potential  benefits 
that  transcend  individual  program  outcomes.  Specifically,  the  ability  to 
execute  a  program  more  predictably  within  cost  and  schedule  estimates 
would  lessen  the  need  to  offset  cost  increases  by  disrupting  other 
modernization  programs. 


Persistent  Problems 
Are  Reinforced  By  the 
Current  Process 


Despite  good  intentions  and  some  progress,  our  ongoing  reviews  of  DOD’s 
major  weapon  system  acquisitions  are  showing  that  significant  reforms 
have  not  yet  been  reflected  in  the  management  and  decision-making  on 
individual  programs.  The  flagship  systems,  as  well  as  many  other  top 
priorities  in  each  of  the  services,  continue  to  cost  significantly  more,  take 
longer  to  produce,  and  deliver  less  than  was  promised. 


The  differences  in  the  practices  employed  by  the  leading  commercial  firms 
and  DOD  reflect  the  different  demands  imposed  on  programs  by  the 
environments  in  which  they  are  managed.  Specific  practices  take  root  and 
are  sustained  because  they  help  a  program  succeed  in  its  environment. 

The  way  success  and  failure  are  defined  for  commercial  and  defense 
product  developments  differs  considerably,  which  creates  a  different  set 
of  incentives  and  evokes  different  behaviors  from  the  people  managing  the 
programs.  Attempts  at  reforming  weapon  acquisitions  have  not  succeeded 
because  they  did  not  change  these  incentives.  All  of  the  participants  in  the 
acquisition  process  play  a  part  in  creating  incentives.  In  this  context,  the 
Office  of  the  Secretary  of  Defense,  the  Congress,  and  the  services’ 
organizations  each  play  a  critical  role  in  getting  the  better  outcomes 
sought  on  major  weapon  system  programs. 


The  basic  management  goal  for  a  weapon  system  program  in  DOD  is 
similar  to  that  of  a  major  commercial  undertaking:  to  develop  and  deliver  a 
product  that  meets  the  customer’s  needs.  The  product  development  efforts 
of  leading  commercial  firms  are  guided  by  achieving  a  sensible  balance 
among  customer  needs,  product  capabilities,  and  investment  resources. 
However,  the  pressures  of  successfully  competing  for  the  funds  to  start 
and  sustain  a  DOD  acquisition  program  make  for  a  much  different 
business  case.  Compared  with  commercial  programs,  the  DOD 
environment  encourages  launching  product  developments  that  embody 
more  technical  unknowns  and  less  knowledge  about  the  performance  and 
production  risks  they  entail.  A  new  weapon  system  is  encouraged  to 
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possess  performance  features  that  significantly  distinguish  it  from  other 
systems.  Consequently,  DOD  program  managers  have  incentives  to 
promote  performance  features  and  design  characteristics  that  rely  on 
immature  technologies.  These  managers  rely  much  more  on  maturing 
technology  during  product  development — when  attention  should  be 
focused  on  design  and  manufacturing — than  do  commercial  programs. 

Even  though  less  information  about  a  new  product  development  is 
available  at  the  time  of  launch,  the  competition  for  funding  requires 
detailed  projections  to  be  made  based  on  this  information.  Product 
development  cannot  be  launched  unless  a  program’s  development  and 
production  cost,  as  well  as  timing,  coincides  with  available  funding. 
Because  DOD  relies  largely  on  forecasts  of  cost,  schedule,  and 
performance  that  are  comparatively  soft,  competing  for  funds  encourages 
the  estimates  to  be  squeezed  into  profiles  of  available  funding.  Additional 
requirements,  such  as  high  reliability  and  maintainability,  serve  to  make 
the  fit  even  tighter.  As  competition  for  funding  will  continue  throughout 
the  program’s  development,  success  becomes  identified  with  the  ability  to 
secure  the  next  installment.  The  risks  associated  with  developing  new 
technologies  and  the  product  together — within  tight  estimates — have 
come  to  be  accepted  as  standard.  Production  realities  are  usually  too 
distant  to  have  a  curbing  effect  on  early  technology  decisions. 

Rewards  for  discovering  and  recognizing  potential  problems  early  in  a 
weapon  system  development  are  few.  Less  available  knowledge  makes  it 
harder  for  program  managers  to  say  “no”  to  unreasonable  expectations. 
Other  factors,  such  as  short  tenures  and  career  pressures,  discourage 
program  managers  from  saying  no.  In  contrast  with  leading  commercial 
firms,  not  having  attained  knowledge — such  as  the  demonstrated  maturity 
of  a  technology  or  a  key  manufacturing  process — can  be  perceived  as 
better  than  knowing  that  problems  exist.  For  these  reasons,  the  practices 
associated  with  successful  commercial  ventures  are  not  readily  adopted  in 
DOD. 


How  a  Knowledge- 
Based  Process  Can 
Lead  to  Better 
Weapons  Acquisitions 


Leading  commercial  firms  expect  that  their  program  managers  will  deliver 
high  quality  products  on  time  and  within  budget.  Doing  otherwise  could 
result  in  the  customer  walking  away.  Thus,  the  firms  have  created  an 
environment  and  adopted  practices  that  put  their  program  managers  in  a 
good  position  to  succeed  in  meeting  these  expectations.  Collectively,  these 
practices  comprise  a  process  that  is  anchored  in  knowledge.  It  is  a  process 
in  which  technology  development  and  product  development  are  treated 
differently  and  managed  separately.  The  process  of  developing  technology 
culminates  in  discovery — the  gathering  of  knowledge — and  must,  by  its 
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nature,  allow  room  for  unexpected  results  and  delays.  Leading  firms  do 
not  ask  their  product  managers  to  develop  technology. 

On  the  other  hand,  product  development  in  commercial  ventures  is  a 
clearly  defined  undertaking  for  which  firms  insist  on  having  in  hand  the 
technology  that  meets  customers’  needs  before  starting.  The  process  of 
developing  a  product  culminates  in  delivery,  and  therefore,  gives  great 
weight  to  design  and  production.  The  firms  demand — and  receive — 
specific  knowledge  about  a  new  product  before  production  begins.  A 
program  does  not  go  forward  unless  a  strong  business  case  on  which  the 
program  was  originallyjustified  continues  to  hold  true.  This  process  is 
depicted  in  figure  1. 


Figure  1:  Knowledge-based  Process  for  Applying  Best  Practices  to  the  Development  of  New  Products 


Such  a  knowledge-based  process  is  essential  to  commercial  firms  getting 
better  cost,  schedule,  and  performance  outcomes.  It  enables 
decisionmakers  to  be  reasonably  certain  about  critical  facets  of  the 
product  under  development  when  they  need  it.  This  knowledge  can  be 
broken  down  into  three  knowledge  points: 

•  when  a  match  is  made  between  the  customer’s  needs  and  the  available 
technology: 

•  when  the  product’s  design  meets  performance  requirements, 

•  and  when  the  product  can  be  produced  within  cost,  schedule,  and  quality 
targets. 

There  is  a  synergy  in  this  process,  as  the  attainment  of  each  successive 
knowledge  point  builds  on  the  preceding  one.  Metrics  gauge  when  the 
requisite  level  of  knowledge  has  been  attained.  The  result  is  a  product 
development  process  that  delivers  excellent  results  in  a  predictable 
manner.  On  the  other  hand,  in  DOD’s  current  acquisition  process, 
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technology,  design,  and  manufacturing  knowledge  is  attained 
concurrently — in  the  higher  cost  environment  of  product  development — 
throughout  the  weapon  system  phases.  This  process  and  the  knowledge 
gathering  that  takes  place  is  shown  in  figure  2. 


Figure  2:  DOD’s  Current  Weapon  System  Acquisition  Cycle 
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We  have  reported  our  findings  on  best  commercial  practices  to  DOD,  and 
have  made  recommendations  on  how  they  can  be  applied  to  weapon 
systems.  A  listing  of  these  reports  is  included  in  the  appendix.  DOD  has 
agreed  with  these  recommendations  and  they  have  been  reflected  in 
DOD’s  draft  revision  of  its  regulations  that  guide  the  acquisition  process. 
We  have  incorporated  some  of  the  terms  from  the  draft  regulations  in  the 
acquisition  process  outlined  in  the  pages  that  follow. 


Technology  and 
Requirements  Must  Match 
Before  a  Program  Can  Be 
Launched 


As  a  technology  is  developed,  it  moves  from  a  concept  to  a  feasible 
invention  to  a  component  that  must  fit  onto  a  product  and  demonstrate 
that  it  can  function  as  expected.  Leading  commercial  firms  do  not  let  a 
new  technology  onto  the  product  development  until  it  reaches  this  level  of 
demonstration.  To  minimize  technology  development  during  product 
development,  leading  commercial  companies  employ  disciplined 
processes  to  match  requirements  with  technological  capability  before  the 
product  development  is  launched.  Prior  to  launching  the  product 
development  program  companies  confront  production  realities  and 
demand  proof  that  the  technology  will  work  and  can  be  successfully 
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produced.  The  companies  bring  solid  technological  knowledge  to  the 
requirements  process  in  the  form  of  current,  valid  information  from 
predecessor  programs,  experienced  people,  and  new  technologies  that 
have  been  demonstrated  to  a  high  level  of  maturity.  In  addition,  the 
companies  communicate  extensively  with  customers  to  match  their  wants 
and  needs  with  the  firm’s  available  technology  and  with  its  ability  to 
manufacture  an  appropriate  product.  They  have  learned  not  to  commit  to 
new  products  that  outstrip  their  technological  know-how.  These  practices 
stem  from  their  recognition  that  resolving  technology  problems  in  product 
development  can  result  in  at  least  a  ten-fold  cost  increase;  resolving  them 
in  production  could  increase  costs  by  a  hundred  fold. 

Other  practices  contribute  to  successfully  developing  technology.  These 
are: 

•  flexibility  in  both  the  resources  provided  and  the  product’s  performance 
requirements  to  allow  for  the  uncertainties  of  technical  progress; 

•  disciplined  paths  for  technology  to  take  toward  inclusion  in  products,  with 
strong  gatekeepers  to  decide  when  to  allow  it  into  a  product  development 
program; 

•  high  standards  forjudging  the  maturity  and  readiness  of  technology; 

•  the  imposition  of  strict  product  development  cycle  times;  and 

•  rules  concerning  how  much  innovation  can  be  accepted  on  a  product 
before  the  next  generation  must  be  launched. 

Collectively,  these  factors  create  a  healthy  environment  for  developing 
technology  and  making  good  decisions  on  what  to  include  in  a  product. 

These  practices  can  be  adapted  to  the  front  end  of  DOD’s  weapon 
acquisition  process.  They  can  relieve  the  pressure  to  launch  programs  with 
immature  technologies  and  underestimated  resources  that  are 
overmatched  by  requirements.  In  fact,  in  some  cases  because  of  unique 
circumstances,  DOD  has  applied  these  practices  with  successful 
outcomes.  For  example,  the  Defense  Advanced  Research  Projects  Agency 
matured  a  revolutionary  periscope  technology  to  a  very  high  maturity  level 
before  it  was  included  on  the  Virginia  class  attack  submarine  development 
program.  To  date,  this  new  technology  has  not  caused  cost  and  schedule 
problems  for  the  submarine’s  development  program.  More  typically, 
however,  DOD  launches  programs  with  immature  technologies  and  does 
not  get  a  match  between  mature  technology  and  the  weapon’s 
requirements  until  late  in  product  development.  For  example,  we  found 
key  technologies  for  the  Army’s  Brilliant  Antiarmor  Submunition  were  at 
very  low  maturity  levels  when  the  acquisition  program  was  launched.  This 
was  a  significant  gap  between  technology  maturity  and  weapon  system 
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requirements  that  was  not  closed  until  well  into  the  program.  Problems 
with  the  technologies  were  a  main  contributor  to  the  program’s  88-percent 
cost  growth  and  62-percent  slip  in  schedule. 

DOD’s  current  acquisition  process  begins  with  a  determination  that  a 
military  need  must  be  met  with  a  new  weapon  system.  Different  concepts 
are  explored  to  determine  the  best  type  of  weapon  system  to  meet  the 
need.  Once  the  concept  has  been  selected,  DOD  typically  launches  the 
development  program  and  assigns  a  program  manager  responsibility  for 
formulating  the  weapon  system  design  and  maturing  essential 
technologies.  DOD  refers  to  this  phase  as  program  definition  and  risk 
reduction.  In  applying  best  practices,  we  have  recommended  that  DOD  not 
launch  a  program  until  the  technologies  needed  to  meet  a  new  weapon’s 
requirements  are  mature.  To  separate  this  technology  development  from 
the  program,  best  practices  suggest  that  a  technology  and  concept 
maturation  phase  follow  concept  exploration  and  precede  program 
launch.  This  and  the  concept  exploration  phase  are  illustrated  in  figure  3. 


Figure  3:  Weapon  Acquisition  Phases  That  Should  Precede  Launching  A 
New  Program 
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Once  the  concept  phase  has  concluded  with  selecting  a  weapon  system 
concept,  such  as  a  satellite,  to  meet  a  need,  the  technology  and  concept 
maturation  phase  would  start.  Practices  included  in  this  phase  are: 

•  assigning  science  and  technology  organizations  responsibility  for 
identifying  key  technologies  and  maturing  them  to  acceptable  levels; 

•  developing  a  preliminary  system  design; 
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•  identifying  enabling  technologies  and  applying  standards  for  managing  and 
assessing  their  maturity  using  technology  readiness  levels1; 

•  using  a  hybrid  organization  or  some  other  means  to  smoothly  transition 
the  responsibility  for  technologies  from  science  and  technology 
organizations  to  the  program  manager,  and 

•  establishing  a  baseline  set  of  weapon  system  requirements,  within 
reasonable  cost  and  schedule  expectations,  that  can  be  met  with  mature 
technologies. 

These  practices  would  separate  technology  development  from  an 
acquisition  program.  They  would  also  result  in  an  established  baseline 
requirement,  a  best  technical  approach,  and  technologies  matured  to  a 
high  standard.  This  would  make  it  possible  to  achieve  a  match  between 
performance  requirements  and  mature  technologies — a  key  knowledge 
point — and  would  enable  decisionmakers  to  not  launch  a  program  if  the 
match  was  not  demonstrated.  Given  a  baseline  design,  decisionmakers 
could  also  defer  unmet  requirements  and  the  associated  technologies  to 
future  upgrades,  allowing  technology  development  to  take  place  in  a  more 
flexible,  failure-tolerant  environment.  Collectively,  these  practices  would 
curb  incentives  to  overpromise  the  capabilities  of  a  new  weapon  system 
and  to  rely  on  immature  technologies.  This  would  put  the  program 
manager  in  a  better  position  to  succeed  once  the  program  is  launched. 


Product  Development 
Should  Deliver  a  Mature 
Design  and  Key  Processes 


The  decision  to  launch  a  major  new  product  development  is  a  watershed 
event  for  leading  commercial  firms.  A  product  development  will  not  be 
launched  unless  the  following  elements  are  present: 

a  clear  market  (customer  requirements)  exists; 

the  firm  possesses  the  technology  and  engineering  capability  to  design  the 
product  and  bring  it  to  market  on  time; 

the  firm  can  afford  the  financial  investment  to  develop  the  product;  and 
the  product  can  be  manufactured  on  time,  at  high  quality,  and  at  a  cost 
that  enables  an  acceptable  return  on  investment. 


If  the  firm  does  not  have  confidence  in  one  or  more  of  these  elements,  it 
does  not  launch  the  product.  On  the  other  hand,  if  all  the  elements  are 
there,  the  manager  of  the  product  development  effort  is  in  a  good  position 


1  In  our  review  of  best  practices  for  including  new  technology  in  products  (GA0/NSIAD-99-162,  July 
1999) ,  we  applied  a  scale  of  technology  readiness  levels  pioneered  by  the  National  Aeronautics  and 
Space  Administration  and  adapted  by  the  Air  Force  Research  Laboratories.  These  levels  have  proven 
to  be  reliable  indicators  of  technology  maturity  and  eventual  success  in  product  development. 
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to  succeed.  The  technology  needed  for  the  product  is  proven,  the  firm  has 
evidence  that  it  can  develop  and  produce  the  product  as  scheduled,  and 
the  venture  can  be  afforded.  Time  constraints  are  an  important  element.  If 
the  product  development  cannot  be  completed  within  a  set  schedule, 
competitors  might  beat  the  firm  to  market  and  the  cost  of  schedule 
overruns  would  hurt  profits.  Time  constraints  are  also  used  to  limit  the 
initial  product’s  requirements.  Another,  perhaps  more  basic,  consideration 
limits  the  product  development  schedule:  the  amount  of  time  one  can 
reasonably  expect  people  to  drive  to  a  goal  like  delivering  a  product.  One 
commercial  executive  observed  that  it  is  unreasonable  to  expect  people  to 
focus  on  a  goal  (such  as  production  start-up)  that  is  4  or  more  years  away. 
To  live  within  these  time  constraints  and  keep  innovation  alive, 
commercial  firms  have  adopted  the  practice  of  saving  requirements  that 
cannot  be  met  with  proven  technologies  for  the  next  generation  product. 

In  DOD’s  current  acquisition  process,  the  decision  point  that  most  closely 
resembles  commercial  firms’  product  development  phase  is  the 
engineering  and  manufacturing  development  decision.  Although  the 
decision  to  launch  the  weapon  system  acquisition  program  has  already 
been  made  years  earlier,  the  engineering  and  manufacturing  development 
phase  represents  DOD’s  effort  to  design,  test,  and  begin  production  of  the 
new  weapon.  Its  broad  scope  places  an  enormous  burden  on  the  program 
manager.  Technologies  still  must  be  matured  and  integrated  into  a  system, 
testing  conducted,  and  production  processes  readied.  Although  this  range 
of  activities  can  take  8  to  10  years,  the  program  manager  is  expected  to 
develop  and  be  accountable  for  a  precise  cost  and  schedule  estimate  made 
at  the  decision  to  go  forward  with  development.  Because  of  their  short 
tenures,  it  normally  takes  several  program  managers  to  complete  this 
phase  alone.  Consequently,  the  program  manager  that  commits  to  the  cost 
and  schedule  estimate  during  the  program  definition  and  risk  reduction 
phase  is  not  the  same  person  responsible  for  achieving  it. 

Best  practices  can  be  applied  to  this  scope  of  activities,  starting  with  the 
premise  that  key  technologies  are  matured  and  matched  to  a  basic  set  of 
user  requirements.  Best  practices  suggest  a  product  development  phase 
consisting  of  two  elements — system  integration  and  system 
demonstration — to  be  completed  in  5  years.  Leading  commercial  firms 
commit  to  product  development  as  a  single  phase  that  combines  both 
elements.  About  midway  through  product  development,  they  become 
certain  of  the  product  design’s  ability  to  meet  customer  requirements, 
indicated  by  having  90  percent  of  engineering  drawings  completed.  In 
addition  to  this  knowledge  point,  they  also  have  all  key  production 
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processes  under  statistical  control  by  the  end  of  product  development — 
the  third  knowledge  point.2  In  applying  best  practices  to  DOD,  there  may 
be  more  discovery — that  is,  more  potential  to  find  the  unexpected — when 
the  individual  components  and  technologies  are  brought  together  as  a 
whole.  Thus,  a  distinct  system  integration  effort  makes  sense.  Similarly, 
DOD  programs  may  also  have  more  discovery  on  production  processes, 
given  the  long  time  spans  between  programs.  Therefore,  key  production 
processes  should  be  demonstrated  by  the  end  of  system  demonstration, 
with  full  process  control  being  achieved  during  low  rate  initial  production. 
Figure  4  illustrates  a  product  development  phase  incorporating  best 
practices. 


Figure  4:  Product  Development  Phase  to  Deliver  a  Mature  Design  and  Key 
Processes 


5-year  Product  Development 


System  Integration 


System  Demonstration 


Program  launch  Design  is  Product  and 

with  mature  mature  processes 

technologies  demonstrated 


The  system  integration  component  of  this  phase  is  the  first  point  in  the 
process  that  vests  full  responsibility  in  the  program  manager,  but 
recognizes  that  a  fair  amount  of  system-level  discovery  can  still  occur 
when  components  and  technologies  are  integrated  into  a  complete 
weapon  system.  Because  of  this,  the  program  manager  should  not  be  held 
to  precise  cost  and  schedule  estimates  to  complete  development  and 
production  because  requisite  knowledge  is  absent.  Therefore,  system 
integration  activity  should  be  accomplished  in  a  flexible  cost  and  contract 
environment.  System  integration  should  be  used  by  the  program  manager 


2  Statistical  process  control  is  a  technique  for  monitoring  production  processes  to  see  if  they  are 
consistently  producing  output  that  is  within  the  quality  standards  and  tolerances  set  for  the  overall 
product. 
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to  gather  the  knowledge  necessary  to  build  firm  cost  and  schedule  targets 
and  to  firm  the  weapon’s  design.  Activities  included  in  this  phase  are: 

•  program  launch; 

•  integration  of  all  technologies  into  a  complete  system; 

•  maturation  of  the  system  design; 

•  completion  of  at  least  90  percent  of  the  system’s  engineering  drawings; 

•  development  of  knowledge-based  cost  and  schedule  estimates;  and 

•  finalization  of  the  system’s  requirements. 

System  integration  should  conclude  with  a  critical  design  review  of 
engineering  drawings  and  confirmation  that  the  system’s  design  will  meet 
requirements — a  key  knowledge  point.  It  should  also  result  in  firm  cost 
and  schedule  targets  and  a  final  set  of  requirements  for  the  initial  version 
of  the  weapon.  Decision-makers  should  insist  on  a  mature  design — 
supported  with  completed  engineering  drawings — before  proceeding  to 
system  demonstration.  Having  such  knowledge  at  this  point  in  time 
contributes  to  product  success. 

For  example,  Boeing  had  released  over  90  percent  of  the  engineering 
drawings  for  its  777-200  airplane  half-way  through  its  product 
development  program.  This  allowed  Boeing  to  have  near  certainty  that  the 
design  for  the  777-200  airplane  would  meet  requirements.  On  the  other 
hand,  DOD’s  C-17  program  had  released  only  56  percent  of  its  engineering 
drawings  at  approximately  the  same  point  in  development.  We  estimate 
that  the  C-17  program  did  not  release  over  90  percent  of  its  initial  drawings 
until  more  than  three  years  later  and  after  7  production  aircraft  had  been 
delivered.  The  C-17  encountered  numerous  technical  problems  in  testing 
that  resulted  in  redesigns,  cost  increases,  and  schedule  delays. 

I  have  mentioned  the  value  of  setting  time  constraints  for  product 
development.  The  leading  commercial  firms  we  have  visited  consciously 
limited  their  development  phases,  ranging  from  18  months  to  just  over  4 
years.  We  have  identified  a  5-year  limit  for  weapon  developments  that 
reflects  both  commercial  practices  and  guidance  provided  by  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  that 
weapons  should  complete  development  and  initial  production  within  7 
years.  This  would  make  it  possible  for  DOD  to  extend  the  tenure  for  a 
single  program  manager  to  the  entire  product  development  phase — 
providing  the  manager  a  more  realistic  responsibility  but  with  more 
accountability.  Accordingly,  the  system  integration  phase  should  last  no 
longer  than  half — 2.5  years — of  the  product  development  phase. 
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System  demonstration  is  the  point  in  product  development  when  the 
program  manager  can  legitimately  commit  to  firm  cost,  schedule,  and 
performance  targets  within  which  to  deliver  the  weapon  system.  In  turn, 
the  program  manager  should  secure  a  full  funding  commitment  from  DOD 
fund  managers.  Included  in  system  demonstration  are: 

•  building  of  weapon  system  prototypes: 

•  test  and  evaluation  of  those  prototypes:  and 

•  test  and  evaluation  of  the  key  processes  used  to  build  them. 

Demonstrating  the  system  should  conclude  with  demonstrations  of  the 
product  in  a  realistic  environment  and  of  the  key  manufacturing  processes 
used  to  build  the  product.  It  should  be  completed  no  later  than  5  years 
after  the  start  of  system  integration.  We  believe  this  activity  would  make 
the  program  manager  and  contractor  accountable  for  a  low-risk  product 
development  phase  with  precise  cost  and  schedule  targets.  It  would  enable 
the  use  of  a  tighter,  less  flexible  contract  that  contains  a  pricing  structure 
for  which  the  contractor  can  be  held  accountable.  At  its  conclusion, 
decisionmakers  could  make  an  accurate,  informed  decision  to  move  from 
product  development  into  production. 


Successful  Product 
Development  Allows  a 
True  Production  Focus 


At  the  conclusion  of  product  development,  leading  commercial  firms 
reach  the  point  at  which  they  know  that  manufacturing  processes  will 
produce  a  new  product  conforming  to  cost,  quality,  and  schedule  targets 
before  they  begin  producing  a  system.  Reaching  this  point  means  more 
than  knowing  the  product  can  be  manufactured;  it  means  that  all  key 
processes  are  under  control,  such  that  the  quality,  volume,  and  cost  of 
their  output  are  proven  acceptable.  Commercial  firms  rely  on  good 
supplier  relationships,  known  manufacturing  processes,  and  statistical 
process  control  to  achieve  this  knowledge  early.  In  fact,  they  have  all  their 
key  processes  under  statistical  process  control  when  production  begins. 
The  ability  to  establish  control  for  key  processes  before  production  begins 
is  the  culmination  of  all  the  practices  employed  to  identify  and  reduce  risk 
and  is  the  final  knowledge  point.  All  of  the  companies  we  visited  agreed 
that  knowledge  about  technology  and  design  early  in  the  process  makes 
the  control  of  processes  possible. 

In  the  current  weapon  acquisition  process,  DOD  breaks  production  into 
two  phases — low-rate  initial  production  and  full-rate  production. 
Ostensibly,  the  purpose  of  low-rate  production  is  to  produce  the  minimum 
quantity  of  a  new  weapon  for  operational  tests  and  establishment  of  a 
production  base  that  allows  for  an  orderly  increase  to  full  production 
rates.  Typically,  however,  development  work  is  not  concluded  before  low- 
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rate  production  is  started  and  thus  both  are  done  concurrently. 
Consequently,  significant  cost  increases  that  cause  schedule  and  other 
problems  are  not  uncommon  in  early  production,  and  production 
processes  are  often  not  able  to  be  statistically  controlled  even  when  full- 
rate  production  begins. 

Applying  best  commercial  practices  to  the  production  of  weapon  systems 
could  relieve  the  pressures  to  complete  product  development  concurrently 
with  low-rate  production.  If  the  product  design  and  key  manufacturing 
processes  are  fully  demonstrated  in  product  development,  the  scope  of 
low-rate  production  can  be  reduced  to  building  operational  test  articles 
and  maturing  the  production  processes.  Low-rate  production  could  safely 
conclude  when  a  product  has  met  user  requirements  in  operational 
conditions  and  manufacturing  processes  are  under  statistical  process 
control — the  third  key  knowledge  point. 

Putting  all  of  the  pieces  together,  DOD  could  adopt  a  weapon  acquisition 
process  that  is  knowledge-based  and  reflects  best  commercial  practices, 
as  shown  in  figure  5. 


Figure  5:  Potential  DOD  Technology  and  Product  Development  Process  Incorporating  Best  Practices 
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This  process  centers  on  the  separation  of  technology  development  from 
product  development  and  the  accumulation  of  technology,  design,  and 
process  knowledge  measured  at  key  junctures.  As  I  noted  earlier,  DOD  is 
currently  rewriting  its  acquisition  policy  to  reflect  these  concepts  among 
many  other  changes.  It  should  be  recognized  that  the  acquisition  offices 
preparing  this  new  policy  cannot  succeed  without  the  agreement  and 
cooperation  of  other  key  communities  that  influence  the  acquisition 
process.  These  include  the  organizations  responsible  for  writing  and 
approving  weapon  system  requirements,  conducting  science  and 
technology,  and  programming  modernization  funds. 


Implications  of  a 
Knowledge-Based 
Acquisition  Process 
for  the  Future 


While  an  acquisition  policy  anchored  in  best  practices  principles  is 
essential  to  getting  better  program  outcomes,  it  is  the  actions  taken  on 
individual  programs  that  determine  whether  real  change  will  occur.  Within 
the  current  process,  actions  are  taken  on  individual  programs  that  run 
counter  to  policy  and  compromise  sound  principles.  Yet  individual 
actions — not  policy — communicate  the  broader  message  of  “what  will 
work”  to  others  in  the  process.  The  real  test  of  a  policy  that  applies  best 
practices  will  be  the  decisions  made  on  individual  program  launch  and 
funding  decisions.  These  decisions  define  what  success  means  in  DOD  and 
what  practices  contribute  to  success.  Decisions  to  advance  or  fund 
programs  that  do  not  have  enough  knowledge  to  meet  best  practices 
standards  send  signals  to  managers  that  not  having  the  necessary  level  of 
knowledge  is  acceptable. 

The  principles  embodied  in  policy  have  proven  difficult  to  apply  on 
individual  programs.  Pressing  circumstances  are  invoked  as  reasons  for 
making  exceptions  for  the  program  at  hand.  Thus,  on  a  case-by  case  basis, 
decisions  to  approve  programs  despite  knowledge  shortfalls  or  their 
failure  to  live  up  to  advertised  expectations  can  be  rationalized  in  a 
number  of  ways:  an  urgent  threat  needs  to  be  met;  a  production  capability 
needs  to  be  preserved;  despite  shortfalls,  the  new  system  is  more  capable 
than  the  one  it  is  replacing;  the  new  system’s  problems  will  be  fixed  in  the 
future.  The  challenge  for  acquisition  participants  is  to  treat  individual 
program  decisions  as  more  than  the  case  at  hand.  They  must  weigh  and  be 
accountable  for  the  broader  implications  of  what  is  acceptable  or  “what 
will  work”  and  be  willing  to  say  no  to  programs  that  run  counter  to  best 
practices.  We  have  made  recommendations  along  these  lines  to  the 
Secretary  of  Defense  and  the  Congress.  Acting  on  them  will  be  key  to  the 
success  of  an  acquisition  policy  anchored  in  best  practices. 

If  DOD  succeeds  in  applying  knowledge-based  practices  to  the  weapons 
acquisition  process,  the  benefits  transcend  the  management  of  individual 
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programs.  Programs  managed  using  best  practices  are  more  likely  to  be 
executed  within  cost  and  schedule  estimates.  This  will  minimize  surprises 
in  the  form  of  cost  and  schedule  increases  that  are  often  accommodated 
by  disrupting  the  funding  of  other  programs.  Without  such  disruptions, 
DOD  can  better  stabilize  its  overall  modernization  effort.  Decisionmakers 
may  be  more  able  to  focus  on  a  balanced  investment  strategy  rather  than 
continue  the  practice  of  making  decisions  about  how  much  each 
program’s  funding  and  quantities  should  be  reduced  so  that  all  programs 
can  continue  to  be  funded. 

We  also  believe  that  the  idea  of  a  phased  investment  strategy  for  a 
program — with  contracts  tailored  for  more  clearly  defined  phases  of  work, 
such  as  technology  maturation,  system  integration,  and  system 
demonstration — can  leave  the  potential  for  competition  open  longer,  both 
across  the  defense  and  nondefense  industries.  Finally,  limiting  product 
development  cycle  times  to  5  years  or  less  would  allow  for  more  frequent 
assimilation  of  new  technologies  into  weapon  systems,  thereby  reducing 
obsolescence  and  allowing  better  management  of  the  industrial  base. 
Reduced  cycle  times,  coupled  with  planned  upgrades,  could  make  for 
more  frequent  and  predictable  work  in  production,  where  contractors  and 
the  industrial  base  can  profit. 


Mr.  Chairman,  this  concludes  my  prepared  statement.  I  would  be  happy  to 
respond  to  any  questions  you  or  other  Members  of  the  Subcommittee  may 
have. 
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Appendix  I 


Related  GAO  Products 


Best  Practices:  Better  Management  of  Technology  Development  Can 
Improve  Weapon  System  Outcomes  (GAO/NSIAD-99-162.  July  30, 1999). 

Defense  Acquisitions:  Best  Commercial  Practices  Can  Improve  Program 
Outcomes  (GAO/T-NSIAD-99-1 16,  Mar.  17, 1999) . 

Defense  Acquisition:  Improved  Program  Outcomes  Are  Possible  (GAO/T- 
NSIAD-98-123,  Mar.  17,  1998). 

Best  Practices:  Successful  Application  to  Weapon  Acquisition  Requires 
Changes  in  POP’s  Environment  (GAO/NSIAD-98-56,  Feb.  24, 1998). 

Best  Practices:  Commercial  Quality  Assurance  Practices  Offer 
Improvements  for  POD  (GAO/NSIAD-96-162,  Aug.  26, 1996). 
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Ordering  Information 


Orders  by  Internet 


For  information  on  how  to  access  GAO  reports  on  the  Internet,  send 
an  e-mail  message  with  “info”  in  the  body  to: 

Info@www.gao.gov 

or  visit  GAO’s  World  Wide  Web  home  page  at: 
http://www.gao.gov 


Contact  one: 

Web  site:  http://www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

1-800-424-5454  (automated  answering  system) 


To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 


